Method and system for human motion recognition

ABSTRACT

A system and method for human motion recognition are provided. The system includes a video sequence decomposer, a feature extractor, and a motion recognition module. The video sequence decomposer decomposes a video sequence into a plurality of atomic actions. The feature extractor extracts features from each of the plurality of atomic actions, the features including at least a motion feature and a shape feature. And the motion recognition module performs motion recognition for each of the plurality of atomic actions in response to the features.

CROSS-REFERENCE TO RELATED APPLICATION

The present application is the U.S. National Stage under 35 U.S.C. §371of International Patent Application No. PCT/SG2014/000275, filed Jun.12, 2014, which claims priority to Singapore Application No. SG201304548-9, filed Jun. 12, 2013, the disclosures of which are herebyincorporated herein by reference.

FIELD OF THE INVENTION

The present invention generally relates to surveillance and recognitiontechnology, and more particularly relates to a system and method forhuman motion recognition.

BACKGROUND OF THE DISCLOSURE

There has been a surge, in recent years, towards the study of humanaction recognition because it is fundamental to many computer visionapplications such as video surveillance, human-computer interface, andcontent-based video retrieval. While the human brain can recognize anaction in a seemingly effortless fashion, recognition solutions usingcomputers have, in many cases, proved to be immensely difficult.

One challenge is the choice of optimal representations for humanactions. Ideally, the representation should be robust against inter- orintra-variations, noises, temporal variations, and sufficiently rich todifferentiate a large number of possible actions. Practically, suchrepresentations do not exist.

It is well documented that human actions can be encoded as spatialinformation of body poses and dynamic information of body motions.However, some actions cannot be distinguished solely using shape and/ormotion features. For example, a skip action may look very similar to arun action if only the pose of the body is observed.

The classification task would be simplified if the motion flow of theentire body is considered simultaneously. Using this approach, one wouldexpect that the skip action generates more vertical flows (upward anddownward flows) than the run action. In addition, actions such asjogging, walking and running can be easily confused if only the poseinformation is used due to the similarity of postures in the actionsequences.

Likewise, there are some actions which cannot be fully described bymotion feature alone. Combining both motion and shape cues potentiallyprovides complementary information about an action. Thus,conventionally, motion and shape feature vectors are concatenated toform a super vector. However, the super vector obtained through suchconcatenation may not explicitly convey the underlying action. Moreover,the super vector is unnecessarily long and requires complex featuredimension reduction techniques.

Thus, what is needed is a system and method for efficient recognition ofhuman motion. Furthermore, other desirable features and characteristicswill become apparent from the subsequent detailed description and theappended claims, taken in conjunction with the accompanying drawings andthis background of the disclosure.

SUMMARY

According to the Detailed Description, a method for human motionrecognition is provided. The method includes decomposing a videosequence into a plurality of atomic actions and extracting features fromeach of the plurality of atomic actions. The features extracted includeat least a motion feature and a shape feature. The method furtherincludes performing motion recognition for each of the plurality ofatomic actions in response to the features.

In accordance with another aspect, a system for human motion recognitionis provided. The system includes a video sequence decomposer, a featureextractor, and a motion recognition module. The video sequencedecomposer decomposes a video sequence into a plurality of atomicactions. The feature extractor extracts features from each of theplurality of atomic actions, the features including at least a motionfeature and a shape feature. And the motion recognition module performsmotion recognition for each of the plurality of atomic actions inresponse to the features.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying figures, where like reference numerals refer toidentical or functionally similar elements throughout the separate viewsand which together with the detailed description below are incorporatedin and form part of the specification, serve to illustrate variousembodiments and to explain various principles and advantages inaccordance with a present embodiment.

FIG. 1, comprising FIGS. 1A to 1D, pictorially illustrates human motionaction sequences, wherein FIG. 1A illustrates action sequences of thehuman motion of walk, FIG. 1B illustrates action sequences of the humanmotion of run, FIG. 1C illustrates the human motion of skip, and FIG. 1Dillustrates the human motion of run for comparison to the human motionof skip (FIG. 1C).

FIG. 2, comprising FIGS. 2A, 2B and 2C, depicts decomposition of humanwalking action in accordance with a present embodiment, wherein FIG. 2Adepicts right-leg stepping, FIG. 2B two-leg crossing, and FIG. 2Cleft-leg stepping.

FIG. 3, comprising FIGS. 3A, 3B and 3C, depicts motion flow and shapeflow vectors of the human motion of running in accordance with thepresent embodiment, wherein FIG. 3A depicts the motion of running withboth vectors, FIG. 3B depicts the motion flow vectors only, and FIG. 3Cdepicts the shape flow vectors only.

FIG. 4, comprising FIGS. 4A and 4B, depicts illustrations of histogrambinning in accordance with the present embodiment, wherein FIG. 4Adepicts a graph of the human motion vectors of FIG. 3 and FIG. 4Bdepicts histogram binning the vectors of FIG. 4A in accordance with thepresent embodiment.

FIG. 5, comprising FIGS. 5A and 5B, depicts regional concatenationhistogram binning in accordance with the present embodiment, whereinFIG. 5A depicts a bounding box illustrating a human motions and dividedinto four regions and FIG. 5B depicts concatenation of resultanthistograms from each region.

FIG. 6 depicts a block diagram of a system for resolving an action videointo a collection of repetitive atomic actions in accordance with thepresent embodiment.

FIG. 7 depicts a representation of a first portion of the operation ofthe system of FIG. 6 where the action video of human motion on the leftis resolved into a combination of regional concatenation of resultanthistograms in accordance with the present embodiment.

FIG. 8 depicts a representation of a second portion of the operation ofthe system of FIG. 6 where the resultant histograms are clustered intodistance weighted bag-of-automatic-actions in accordance with thepresent embodiment.

FIG. 9, comprising FIGS. 9A, 9B and 9C, depicts motion recognition inaccordance with the present embodiment of six different human actionsequences from two different human action sequence datasets, whereinFIG. 9A illustrates the six different human action sequences utilizedfor the comparison, FIG. 9B is a graph of the accuracy of recognition ofvarious human action features in a first human action sequence datasetby motion recognition in accordance with the present embodiment, andFIG. 9C is a graph of the accuracy of recognition of the various humanaction features in a second human action sequence dataset by motionrecognition in accordance with the present embodiment.

And FIG. 10 illustrates a 6×6 matrix of the motion recognition inaccordance with the present embodiment of the six different human actionsequences in both human action sequence datasets of FIG. 9 where the sixdifferent human action sequences are listed along the vertical and therecognized human actions are listed along the horizontal.

Skilled artisans will appreciate that elements in the figures areillustrated for simplicity and clarity and have not necessarily beendepicted to scale. For example, the dimensions of some of the elementsin the block diagrams or flowcharts may be exaggerated in respect toother elements to help to improve understanding of the presentembodiments.

DETAILED DESCRIPTION

The following detailed description is merely exemplary in nature and isnot intended to limit the invention or the application and uses of theinvention. Furthermore, there is no intention to be bound by any theorypresented in the preceding background of the invention or the followingdetailed description. It is the intent of this invention to present anefficient and recognition of human action with improved accuracy.

As stated above, human actions can be encoded as spatial information ofbody poses and dynamic information of body motions. Referring to FIG. 1,comprising FIGS. 1A to 1D, action sequences of human actions are shown.In FIG. 1A, an action sequence 100 depicts five views 102, 104, 106,108, 110 of a person walking. In FIG. 1B, an action sequence 120 depictsfive views 122, 124, 126, 128, 130 of a person running. Some actionscannot be distinguished solely using shape and/or motion features. InFIGS. 1C and 1D, two views 140, 150 depict single body poses of twodifferent human actions. It is difficult to distinguish a skip action inthe view 140 from a run action in the view 150 because when only thepose of the body is observed at a single point in the action, the skipaction view 140 appears very similar to the run action view 150.

In accordance with the present embodiment, a complex human actionsequence is decomposed into a sequence of elementary building blocks,known as ‘atomic actions’. Referring to FIG. 2, comprising FIGS. 2A, 2Band 2C, the human action of walking is broken down into several atomicactions 200, 210, 220. FIG. 2A depicts the atomic action 200 ofright-leg stepping, FIG. 2B depicts the atomic action 210 of two-legcrossing, and FIG. 2C depicts the atomic action 220 left-leg crossing.

Shape and motion are the two most important cues for actions, and atomicactions can be ‘synthesized’ from both elements. FIG. 3, comprisingFIGS. 3A, 3B and 3C, illustrates these cues. In the view 300 of FIG. 3Aan atomic action of a person 302 running is shown. A first set ofvectors 304 are vectors depicting motion flow (e.g., right forearmmoving up, left forearm moving forward, left leg moving forward, rightlower leg moving up). The first set of vectors 304 are extracted into amotion flow vector view 310 in FIG. 3B. A second set of vectors 306 arevectors depicting shape flow (i.e., vectors identifying the outline ofthe shape). The second set of vectors 306 are extracted into a shapeflow vector view 320 in FIG. 3B.

Observing shape and motion is a very natural way to recognize an action.The visual cortex in the brain has two pathways to process shape andmotion information. Motivated by the robustness of histograms offeatures, in accordance with a present embodiment ahistogram-of-oriented gradient (HOOG) and a histogram-of-orientedoptical flow (HOOF) are used as shape and motion descriptors,respectively. HOOG is also be used as a pose descriptor.

Such a feature is more robust against scale variation and the change ofmotion direction. A method for extraction of the HOOF and the HOOG inaccordance with the present embodiment is illustrated in FIG. 4,comprising FIGS. 4A and 4B. In FIG. 4A, a graph 400 of human motionvectors includes a horizontal axis 405 and a vertical axis 410. Pose andshape vectors 412, 414, 416, 418 plotted on the graph 400 and aredivided in accordance with their primary angles from the horizontal axis405 (i.e., their symmetry about the vertical axis 410).

Referring to FIG. 4B, a four-bin histogram 450 bins the vectors 412,414, 416, 418 in accordance with their symmetry about the vertical axis410—that is the vectors 412 are binned in bin 452, the vectors 414 arebinned in bin 454, the vectors 416 are binned in bin 456, and thevectors 418 are binned in bin 458. In this manner, the histogram bargraph 450 depicts human motion feature extraction of pose, shape andmotion as a histogram combination of the HOOG and the HOOF. Thehistogram 450 is normalized to sum up to unity to make itscale-invariant. In this manner, there is no need to normalize the sizeof each of the bounding boxes 452, 454, 456, 458.

As a result, the histogram of a person moving from left to right will bethe same as a histogram of a person moving from right to left (i.e., themethod in accordance with the present embodiment is directionindiscriminate). The contribution of each vector is proportional to itsmagnitude and the histogram is normalized to sum up to unity to make itscale-invariant.

FIG. 5, comprising FIGS. 5A and 5B, depicts regional concatenationhistogram binning in accordance with the present embodiment. Spatialinformation is considered by dividing a bounding box 500 of a subject510 into a 4×4 matrix of regions 512, 514, 516, 518, as shown in FIG.5A. A resultant histogram 550 depicts concatenation of four-binhistograms 552, 554, 556, 558 from each region.

As discussed above in regards to FIG. 2, a complex action can bedecomposed into a sequence of elementary building blocks known as atomicactions. Referring to FIG. 6, a block diagram 600 depicts a system inaccordance with the present embodiment. An atomic action is defined asthe action performed at video frame t of a video 602. It is representedby a shape histogram (i.e., HOOG) 604 extracted at frame t and anoptical flow histogram (i.e., HOOF) 606 computed at frames (t−1) and t.Therefore, the T-frame action video 602 has (T−1) number of atomicactions.

Suppose action, shape, and motion are three discrete random variables:Z; S; M with distribution z[x]; s[x]; and m[x] respectively, where [ ]represents discrete data. s[x] and m[x] are basically the shape andmotion histograms computed. In a further assumption, an action is afunction of shape and motion, i.e., Z=f(S;M). The simplest functionwould be a summation:Z=S+M  (1)

According to probability theory, the sum of discrete random variableswill produce a new random variable with distribution that can bedetermined via convolution. Therefore, the distribution (histogram) ofan action can be determined by

$\begin{matrix}{{z\lbrack x\rbrack} = {{{s\lbrack x\rbrack}*{m\lbrack x\rbrack}} = {\sum\limits_{k = {- \infty}}^{k = {+ \infty}}\;{{s\lbrack k\rbrack} \cdot {m\left\lbrack {x - k} \right\rbrack}}}}} & (2)\end{matrix}$where the asterisk ‘*’ denotes the convolution operator. The idea ofusing a convolution operator is also inspired by success ofconvolution-based reverb applications in digital signal processing(DSP). In DSP, convolution is a mathematical way of combining two sourcesignals to form an output signal. The output signal bears thecharacteristics of both sources. Convolution-based reverb is a processfor digitally simulating the reverberation of a virtual or physicalspace. Given the impulse response of a space which can be obtained byrecording a short burst of a broadband signal, any “dry” signal (littleroom or space influence) can be convolved with the impulse response. Theresult is that the sound appears to have been recorded in that space.Analogously, knowing that an action is characterized by both shape andmotion information, an atomic action histogram can be obtained byconvolving the corresponding shape histogram (HOOG) 604 and motionhistogram (HOOF) 606. The convolution operation 608 is commutative,which means that the order of the inputs does not mathematically matter.

The length of the output is given by the expression ∥s∥+∥m∥−1. Thisrepresentation has two major advantages. First, the action histogram ismore robust against noises. This is because each bin in the actionhistogram is influenced by bins in the shape histogram weighted by themotion histogram or vice versa (the commutative property ofconvolution). Therefore the effect of abrupt changes in the histogrammagnitude can be minimized. Second, the action histogram produced usingconvolution is more discriminative. The ratio of inter-class distance tointra-class distance is measured and the results on a known human actionvideo dataset is shown below in Table 1.

Table 1 shows a comparison of normalized inter-/intra-class distanceratio on a known human action video dataset for different types offeature combination methods where a Hellinger distance measure is usedto compare two histograms:

$\begin{matrix}{{D_{h}\left( {{s\lbrack x\rbrack},{m\lbrack x\rbrack}} \right)} = \left\lbrack {1 - {\sum\limits_{\forall x}\;\sqrt{{s\lbrack x\rbrack} \cdot {m\lbrack x\rbrack}}}} \right\rbrack^{\frac{1}{2}}} & (3)\end{matrix}$A higher value indicates that the feature is potentially morediscriminative.

TABLE 1 Combination Strategies Ratio Convolution (Conv) 1.0000 Summation(Sum) 0.8535 Product (Prod) 0.8489 Concatenation (Concat) 0.8743The results suggest that the convolution operation produces featurevectors that are potentially more discriminative than the featuresobtained through other combination methods.

In one example, an action video is represented as a collection ofrepetitive atomic actions. The basic concept is illustrated in FIG. 6.Recall that atomic actions are characterized by the convolvedshape-motion histogram. A visual codebook can be created by performingK-means clustering on all atomic actions from the training data. Thecluster centroids serve as the visual codewords. Next, each atomicaction in the video is compared against those codewords and thedistances are recorded accordingly. The distance between the atomicaction and its nearest codeword is used to weight the histogram bin. Thehistogram for all relevant codewords in a video is computed byaggregating their respective distances. This final representation allowsany lengthy video to be ‘compressed’ into a compact histogram. Thehistogram is normalized such that a sum of the bins is unity. Thenormalization ensures that the histogram distribution is invariant tothe video length. For instant, given a particular action class,codewords (i.e., key atomic actions) frequencies for a variable lengthvideo remain relatively stable. The aforementioned compact videorepresentation can also be called distance weightedbag-of-automatic-actions (i.e., bag-of-words model 610).

Referring to FIG. 7, another representation 700 of the system 600 (FIG.6) takes an atomic action 702 and extracts a HOOG histograph 704 and aHOOF histograph 706. The histographs 704, 706 are convoluted to form adistance weighted bag-of-automatic-actions resultant histograph 708.

Referring to FIG. 8, a representation 800 of a second portion of theoperation of the system 600 where the resultant histograms are clusteredinto distance weighted bag-of-automatic-actions in accordance with thepresent embodiment is depicted. A human action is broken down intoseveral atomic actions 802. Histograms of the atomic actions 802 areclustered with a K-means operation as shown in a graph 804 of theclusters. Then, each atomic action 802 is assigned to its closestcluster center 806, 808, 810. In this manner, a contribution of eachatomic action 802 to a histogram 820 is inversely proportional to theatomic action's distance from a cluster center. The entire videosequence is then represented in the distance weighted occurrencehistogram 820 of visual words in bins 822, 824, 826, 828, 830. At anormalization step 840, a sum of the bins 822, 824, 826, 828, 830 isnormalized to one. Thus, the sum of the bins 822, 824, 826, 828, 830 isnormalized to one is unity and, thus, is advantageously invariant to avideo length of the entire video sequence of the atomic actions 802.

The action recognition framework in accordance with the presentembodiment has been evaluated using a first and a second publiclyavailable dataset, identified as the Weizmann dataset (the firstdataset) and the KTH dataset (the second dataset). The KTH dataset hasbeen regarded either as one large set with strong intra-subjectvariations (all-in-one) or as four independent scenarios. In the lattercase, each scenario is trained and tested separately. For the KTH-basedevaluation, the focus was on the KTH all-in-one case.

Since the KTH dataset size is much larger than the Weizmann datasetsize, a K-means algorithm is used to cluster the training data as seenin the graph 804. Each class in the KTH dataset is quantized into fivehundred clusters. This quantization can reduce the intra-class variationand computational time. A leave-one-out cross validation (LOOCV)protocol is used in all of the evaluations. Table 2 shows the LOOCVrecognition rate.

TABLE 2 Table 2. LOOCV classification accuracy using different number ofclusters. No. of Dataset Clusters Convol Sum Prod Concat HOOG HOOFWeizmann 5 96.67 87.78 87.78 86.67 66.67 73.33 10 100 94.44 94.44 93.3374.44 78.89 15 100 97.78 94.44 98.89 77.78 81.11 20 100 97.78 96.6797.78 78.89 84.44 25 100 98.89 96.67 97.78 81.11 83.33 KTH 10 83.9470.25 72.22 75.58 45.88 57.73 25 91.63 79.94 79.92 83.30 51.90 63.88 4092.46 82.44 81.43 87.64 45.24 64.88 55 91.46 84.43 83.62 86.97 45.5863.37For the Weizmann dataset which only uses five clusters (codewords), theconvolved feature yields a much higher accuracy (96.67%) as compared toother features. When the number of clusters is increased further, theconvolved feature consistently gives perfect classification accuracy(100%). Using only shape feature (HOOG) or only motion features (HOOF)results in poorer results than using a method in accordance with thepresent embodiment. On average, the method in accordance with thepresent embodiment provided about 11.29% overall improvement as comparedto other methods.

Referring to FIG. 9, comprising FIGS. 9A, 9B and 9C, motion recognitionin accordance with the present embodiment of human action sequences 900including six different human action sequences 902, 904, 906, 908, 910,912 from the Weizmann and KTH human action sequence datasets isperformed, examined and compared. In FIG. 9A, the six different humanaction sequences utilized for the comparison are boxing 902, clapping904, waving 906, walking 908, jogging 910, and running 912. FIG. 9B is agraph 920 of the accuracy (plotted along a vertical axis 924) ofrecognition of various human action features (plotted along a horizontalaxis 922) in the Weizmann human action sequence dataset by motionrecognition in accordance with the present embodiment using a variety ofK-means clustering operations. Traces plotted on the graph 920 showclustering with a K-means operation where K is equal to five 930, whereK is equal to ten 932, where K is equal to fifteen 934, where K is equalto twenty 936, and where K is equal to twenty-five 938. FIG. 9C is agraph 940 of the accuracy (plotted along a vertical axis 944) ofrecognition of various human action features (plotted along a horizontalaxis 942) in the KTH human action sequence dataset by motion recognitionin accordance with the present embodiment, also using a variety ofK-means clustering operations. Traces plotted on the graph 940 showclustering with a K-means operation where K is equal to ten 950, where Kis equal to twenty-five 952, where K is equal to forty 954, and where Kis equal to fifty-five 956.

Higher accuracies are attained from the convolved feature for all numberof clusters of the KTH dataset. The advantage of using the convolvedfeature is more prominent in the KTH dataset. The average improvementover all other five features is 19.56%. Again, the HOOG feature alone orthe HOOF feature alone fails to provide discriminative information. Oneimportant observation from the results in the graphs 920, 940 is thatthe method and system in accordance with the present embodimentconsistently requires a much smaller number of clusters or codewords togive higher accuracy. For example, with only ten clusters, operation inaccordance with the present embodiment achieves comparable accuracy witha product feature which uses forty clusters. This confirms the findingthat the convolved feature is significantly more discriminative thanconventional features.

Referring to FIG. 10, results of the operations of FIG. 9 are summarizedin a 6×6 matrix 1000 of the motion recognition in accordance with thepresent embodiment of the six different human action sequences 902, 904,906, 908, 910, 912 in the KTH human action sequence dataset. The sixdifferent human action sequences are listed along the vertical 1002 andthe recognized human actions are listed along the horizontal 1004. Itcan be seen that human motion recognition for walking, jogging andrunning are in the 90^(th) percentile with the only misclassificationsin these same three actions. Classifications for handclapping are alsoin the 90^(th) percentile with the only misclassifications falling inhandwaving and boxing. While misclassifications for boxing andhandwaving are more plentiful, some of these misclassifications may bedue to the KTH human action sequence dataset in that some of the atomicactions extracted may include an erroneous bounding box which isextracted off-centered from a body axis of the subject in the boundingbox.

In the various examples illustrated above, a method to encode humanactions by convolving shape-motion histograms has been presented. Themain idea is to produce an output signal (i.e., an action histogram)from the source signals (i.e., shape and motion histograms) so that theoutput shares the characteristics of both source signals and inputs. Thefeatures are also much more discriminative than other hybrid featuresobtained through other combination strategies such as concatenation,sum, and product. Further, combination of shape and motion featuresgreatly improves the classification results.

In addition, operation in accordance with the present embodiment avoidsthe need to determine weights manually during feature concatenation. Theconvolved feature is also very compact and has much lower dimensionality(79-dimensional) as compared to conventional concatenated features of512-dimensional and 1000-dimensional methodologies. Due to thediscriminative nature of the convolution feature, the codebook size isextremely small as compared to conventional methods. Also, the entirevideo sequence is advantageously represented as a distance weightedoccurrence histogram of visual words.

Thus, it can be seen that a system and method for human motionrecognition has been provided. The system includes a video sequencedecomposer 602, a feature extractor (including HOOG 604 and HOOF 606),and a motion recognition module (including combiner 608 and bag-of-wordsmodel module 610). The video sequence decomposer decomposes a videosequence into a plurality of atomic actions. The feature extractorextracts features from each of the plurality of atomic actions, thefeatures including at least a motion feature and a shape feature. Andthe motion recognition module performs motion recognition for each ofthe plurality of atomic actions in response to the features.

The motion recognition module performs motion recognition for each ofthe plurality of atomic actions by convolving histograms of the featuresof each of the plurality of atomic actions. In regards to the shapefeature, the feature extractor extracts a set of shape vectors depictingshape flow from each of the plurality of atomic actions and the motionrecognition module convolves histograms of the shape features of each ofthe plurality of atomic actions by deriving a shape descriptor bydetermining a histogram-of-oriented gradient of the set of shape vectorsfor each of the plurality of atomic actions.

In regards to the motion feature, the feature extractor extracts a setof motion vectors depicting motion flow from each of the plurality ofatomic actions and the motion recognition module convolves histograms ofthe motion features of each of the plurality of atomic actions byderiving a motion descriptor by determining a histogram-of-orientedoptical flow of the set of motion vectors for each of the plurality ofatomic actions.

The features may also include a pose feature, and the feature extractorfurther extracts a set of pose vectors from each of the plurality ofatomic actions and the motion recognition module convolves histograms ofthe pose features of each of the plurality of atomic actions by derivinga pose descriptor by determining a histogram-of-oriented gradient of theset of pose vectors for each of the plurality of atomic actions. Thefeatures may also include a spatial feature, and the feature extractorderives each of a set of shape, motion or pose vectors for each of twoor more regions of a bounding box within each of the plurality of atomicactions. The bounding box in each of the plurality of atomic actions isconfigured to include all of a subject pictured in the one of theplurality of atomic actions. The motion recognition module convolveshistograms of each of the shape, motion or pose descriptors to generatea resultant histogram.

The motion recognition module also normalizes the histograms of each ofthe plurality of atomic actions to sum up to unity and further mayinclude a bag-of-words model module for K-means clustering of all of theatomic actions to generate a distance weighted bag-of-automatic-actionsmodel of the video sequence.

Thus, in accordance with the present embodiment an efficient humanmotion recognition system and method is provided. The present embodimentis computationally efficient as compared to conventional motionrecognition systems and even in comparison to conventional combinationstrategies such as sum, product and concatenation. The technology of thepresent embodiment and its various alternates and variants can be usedfor many scenarios. For example, the present embodiment provides acomputationally efficient system and method for many computer visionapplications such as video surveillance, human-computer interface, andcontent-based video retrieval which is robust against inter- orintra-variations, noises, temporal variations, and sufficiently rich todifferentiate a large number of possible actions.

Thus, it can be seen that a system and method for human motionrecognition which reduces complexity of the recognition methodology hasbeen provided. While exemplary embodiments have been presented in theforegoing detailed description of the invention, it should beappreciated that a vast number of variations exist.

It should further be appreciated that the exemplary embodiments are onlyexamples, and are not intended to limit the scope, applicability,operation, or configuration of the invention in any way. Rather, theforegoing detailed description will provide those skilled in the artwith a convenient road map for implementing an exemplary embodiment ofthe invention, it being understood that various changes may be made inthe function and arrangement of elements and method of operationdescribed in an exemplary embodiment without departing from the scope ofthe invention as set forth in the appended claims.

What is claimed is:
 1. A method for human motion recognition comprising:decomposing a video sequence into a plurality of atomic actions;extracting features from each of the plurality of atomic actions, thefeatures comprising at least a motion feature and a shape feature; andperforming motion recognition for each of the plurality of atomicactions in response to the features, wherein the step of performingmotion recognition for each of the plurality of atomic actions comprisesperforming motion recognition for each of the plurality of atomicactions by convolving histograms of the features of each of theplurality of atomic actions.
 2. The method in accordance with claim 1wherein the step of extracting features from each of the plurality ofatomic actions comprises extracting a set of shape vectors depictingshape flow from each of the plurality of atomic actions.
 3. The methodin accordance with claim 2 wherein convolving histograms of the featuresof each of the plurality of atomic actions comprises deriving a shapedescriptor by determining a histogram-of-oriented gradient of the set ofshape vectors for each of the plurality of atomic actions.
 4. The methodin accordance with claim 1 wherein the step of extracting features fromeach of the plurality of atomic actions comprises extracting a set ofmotion vectors depicting motion flow from each of the plurality ofatomic actions.
 5. The method in accordance with claim 4 whereinconvolving histograms of the features of each of the plurality of atomicactions comprises deriving a motion descriptor by determining ahistogram-of-oriented optical flow of the set of motion vectors for eachof the plurality of atomic actions.
 6. The method in accordance withclaim 1 wherein the features further comprise a pose feature, andwherein the step of extracting features from each of the plurality ofatomic actions comprises extracting a set of pose vectors from each ofthe plurality of atomic actions, and wherein convolving histograms ofthe features of each of the plurality of atomic actions comprisesderiving a pose descriptor by determining a histogram-of-orientedgradient of the set of shape vectors for each of the plurality of atomicactions.
 7. The method in accordance with claim 1 wherein the featuresfurther comprise a spatial feature, and wherein the step of extractingfeatures from each of the plurality of atomic actions comprises derivingeach of a set of shape, motion or pose vectors for each of two or moreregions of a bounding box within each of the plurality of atomicactions, the bounding box in each of the plurality of atomic actionsconfigured to include all of a subject pictured in the one of theplurality of atomic actions, and wherein the step of performing motionrecognition for each of the plurality of atomic actions comprisesconvolving histograms of each of the shape, motion or pose descriptorsto generate a resultant histogram.
 8. The method in accordance withclaim 1 wherein convolving histograms of the features of each of theplurality of atomic actions comprises normalization of the histograms ofeach of the plurality of atomic actions to sum up to unity.
 9. Themethod in accordance with claim 1 further comprising K-means clusteringof all of the atomic actions to generate a distance weightedbag-of-automatic-actions model of the video sequence.
 10. A system forhuman motion recognition comprising: a video sequence decomposer fordecomposing a video sequence into a plurality of atomic actions; afeature extractor for extracting features from each of the plurality ofatomic actions, the features comprising at least a motion feature and ashape feature; and a motion recognition module for performing motionrecognition for each of the plurality of atomic actions in response tothe features, wherein the motion recognition module performs motionrecognition for each of the plurality of atomic actions by convolvinghistograms of the features of each of the plurality of atomic actions.11. The system in accordance with claim 10 wherein the feature extractorextracts a set of shape vectors depicting shape flow from each of theplurality of atomic actions.
 12. The system in accordance with claim 11wherein the motion recognition module convolves histograms of the shapefeatures of each of the plurality of atomic actions by deriving a shapedescriptor by determining a histogram-of-oriented gradient of the set ofshape vectors for each of the plurality of atomic actions.
 13. Thesystem in accordance with claim 10 wherein the feature extractorextracts a set of motion vectors depicting motion flow from each of theplurality of atomic actions.
 14. The system in accordance with claim 13wherein the motion recognition module convolves histograms of the motionfeatures of each of the plurality of atomic actions by deriving a motiondescriptor by determining a histogram-of-oriented optical flow of theset of motion vectors for each of the plurality of atomic actions. 15.The system in accordance with claim 10 wherein the features furthercomprise a pose feature, and wherein the feature extractor furtherextracts a set of pose vectors from each of the plurality of atomicactions, and wherein the motion recognition module convolves histogramsof the pose features of each of the plurality of atomic actions byderiving a pose descriptor by determining a histogram-of-orientedgradient of the set of pose vectors for each of the plurality of atomicactions.
 16. The system in accordance with claim 10 wherein the featuresfurther comprise a spatial feature, and wherein the feature extractorderives each of a set of shape, motion or pose vectors for each of twoor more regions of a bounding box within each of the plurality of atomicactions, the bounding box in each of the plurality of atomic actionsconfigured to include all of a subject pictured in the one of theplurality of atomic actions, and wherein the motion recognition moduleconvolves histograms of each of the shape, motion or pose descriptors togenerate a resultant histogram.
 17. The system in accordance with claim10 wherein the motion recognition module normalizes the histograms ofeach of the plurality of atomic actions to sum up to unity.
 18. A systemfor human motion recognition comprising: a video sequence decomposer fordecomposing a video sequence into a plurality of atomic actions; afeature extractor for extracting features from each of the plurality ofatomic actions, the features comprising at least a motion feature and ashape feature; and a motion recognition module for performing motionrecognition for each of the plurality of atomic actions in response tothe features, wherein the motion recognition module further comprises abag-of-words model module for K-means clustering of all of the atomicactions to generate a distance weighted bag-of-automatic-actions modelof the video sequence.